Optimal encoding in stochastic latent-variable Models

We examine the problem of optimal sparse encoding of sensory stimuli by latent variables in stochastic models. Analyzing restricted Boltzmann machines with a communications theory approach, we search for the minimal model size that correctly conveys the correlations in stimulus paŠerns in an information-theoretic sense. We show that the Fisher information Matrix (FIM) reveals the optimal model size. In larger models the FIM reveals that irrelevant parameters are associated with individual latent variables, displaying a surprising amount of order. For well-€t models, we observe the emergence of statistical criticality as diverging generalized susceptibility of the model. In this case, an encoding strategy is adopted where highly informative, but rare stimuli selectively suppress variability in the encoding units. Œe information content of the encoded stimuli acts as an unobserved variable leading to criticality. Together, these results can explain the stimulus-dependent variability suppression observed in sensory systems, and suggest a simple, correlation-based measure to reduce the size of arti€cial neural networks. Signi€cance Currently liŠle is known about the statistical structure of representations in stochastic latent encoders, which serve as models of neuronal sensory systems and are widely used in machine learning. Using approaches from statistical physics and information theory, we show that it is possible to evaluate an optimal size of the latent space, and observe emergence of statistical criticality at this model size. Criticality corresponds to an encoding strategy for handling variable information-content of stimuli in a stochastic channel with a €xed hidden-layer size. Œese results yield testable hypotheses about encoding in neuronal sensory systems, and suggest strategies for improving machine learning models. ar X iv :1 80 2. 10 36 1v 1 [ qbi o. N C ] 2 8 Fe b 20 18